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Abstract — This paper presents an extensible and reusable 
framework which addresses the problem of video quality as- 
sessment over IP networks. The proposed tool (referred to as 
Video-Tester) supports raw uncompressed video encoding and 
decoding. It also includes different video over IP transmission 
methods (i.e.: RTP over UDP unicast and multicast, as well as 
RTP over TCP). In addition, it is furnished with a rich set of 
offline analysis capabilities. Video-Tester analysis includes QoS 
and bitstream parameters estimation (i.e.: bandwidth, packet 
inter-arrival time, jitter and loss rate, as well as GOP size and I- 
frame loss rate). Our design facilitates the integration of virtually 
any existing video quality metric thanks to the adopted Python- 
based modular approach. Video-Tester currently provides PSNR, 
SSIM, ITU-T G.I070 video quality metric, DIV and PSNR-based 
MOS estimations. In order to promote its use and extension, 
Video-Tester is open and publicly available. 

Index Terms — Performance evaluation, objective and subjec- 
tive measures, traffic and performance monitoring, networking 
and QoS, IPTV & Internet TV. 



I. Introduction 

During the last years, there has been a continuous growth 
of video traffic over packet-switched networks [1]. Users' 
demand is determinant in the proliferation of a wide range 
of video-based services and applications such as video on 
demand (VoD), live streaming, video calling, video instant 
messaging, video monitoring, webcam traffic, Internet video 
to TV, live Internet TV, mobile video, etc. All these hetero- 
geneous services are designed to fulfill different purposes and 
requirements [2]. In spite of their dissimilarities, they all share 
a common need. Namely, to make use of tools for video quality 
evaluation. 

The issue of video quality assessment (by defining the 
proper metrics) is of paramount importance for codec design. 
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different transmission methods evaluation and network plan- 
ning among others. Indeed, this problem has generated a lot 
of contributive research efforts [3] along with the specification 
of several video quality measurement standards [4]. 

However, public available tools that systematize the video 
quality assessment over IP networks are scarce. In this respect, 
it is also valuable that the designed assessment tool will 
be highly extensible and reusable since it eases the quick 
inclusion of any metric. 

In this work, we propose Video-Tester, an open-source 
modular framework [5] which provides a complete and flexible 
solution to the problem of video quality assessment over IP 
networks. Video quality evaluation is a multidimensional pro- 
blem that requires to take into consideration multiple criteria, 
sometimes located at different levels. Our tool works at any 
of the three layers involved in any video over IP service — the 
bitstream level, the packet level and the upper picture level 
[6] — . As a remarkable benefit, Video-Tester is able to support 
any existing and future video quality metric. 

The rest of the paper is organized as follows. Section II 
provides the state of the art of the addressed problem. Section 
III summarizes the proposed framework. It explains our Video- 
Tester design and also reviews the current supported metrics. 
Section IV outlines some experimental results that validate 
the proposed framework and, finally. Section V concludes the 
paper and discusses future work and extensions. 

II. Related work 

Several quality evaluation toolboxes have been recently 
proposed, like the implemented by Murthy and Karam [7]. 
However, in general, these applications are mainly restricted 
to codec testing and metric comparisons, because they do not 
consider multimedia transmission methods and the degradation 
and quality impact derived from this process. 

On the other hand, it is also remarkable the seminal work 
(EvalVid) proposed by Klaue et al. [8], a widely used tool- 
set for video transmission evaluation. This tool-set does ac- 
tually include transmission, some quality of service (QoS) 
parameter estimation (packet loss, delay and jitter) and some 
bitstream parameter evaluations (frame loss rate and frame 
jitter). EvalVid video quality evaluation techniques include 
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Fig. 1. Operation scheme. 



PSNR, Structural Similarity index (SSIM) [9], Mean Opinion 
Score (MOS, mapped from PSNR) and Distortion in Inverval 
(DIV, mapped from MOS as described in [10]). 

However, this is a multi-tool framework that requires 3rd 
party tools like tcpdump (packet capture), jfmpeg (video en- 
coding and decoding), netcat and MP4Box [11] (video multi- 
plexation). This fact makes the automatization of its use a bit 
difficult. Furthermore, the use of MP4 multimedia containers 
restricts the available codecs. Added to that, the election of 
ISO-C programming language is definitively appropriate for 
code efficiency (not so for porting because of the necessary 
3rd party tools), but if new quality metrics are needed, the 
source code must be modified. 

Additionally, a common impairment in video transmission 
evaluation is related to frame losses. The decoded RAW video 
at the receiver end will have fewer frames than the original 
one if packet losses occur. As a consequence, we have to deal 
with frame misalignment in order to apply picture metrics like 
PSNR. In [8], it is stated that Fix- Video (FV) tool solves this 
problem. Nevertheless, to the author's knowledge the actual 
version of this framework [ 12] does not include the FV tool. 

III. Proposed framework 

Video-Tester comprises a single command-line application 
that works with a number of configuration files that ease its 
script-based execution. Once Video-Tester is launched at the 
two particular end-points, one side acts as the server and the 
other one as the client. The client side can optionally be 
launched with a user-friendly graphical user interface (GUI) 
in order to plot metrics automatically. 

Video-Tester is written in Python in order to promote its 
extensibility. Video processing (specifically encoding, deco- 
ding and transmission) is performed by using the valuable 
GStreamer library [13] due to its broad and sustained support 
by the research community. As a beneficial consequence, the 
codec support is subjected to the GStreamer support and, 
hence, it is not constrained to a given container format. 

A. Video-Tester design and operation 

Figure 1 shows the general operation scheme. The server 
side is concurrent and talks to clients using the XML-RPC 



protocol. Both the server and the client are able to work behind 
a NAT router (with port redirection at the server side). Each 
client is able to select the following parameters in order to 
start a new test: 

• The uncompressed video file to transmit, selected from 
a common local database (Video-Tester accepts lossless 
encoded videos like [14]). 

. The codec (H.263 [15], H.264 [16], MPEG-4 Visual [17] 
and Theora [18] are currently available). 

• The bitrate (not all bitrates are available with some 
codecs). 

« The frame rate. 

• The transmission method (RTP over UDP unicast or 
multicast, and RTP over TCP are supported). 

Then, the server side sets up a GStreamer-based RTSP 
server (see [19]) instance with the above parameters. At the 
client side, a GStreamer-based client receives the video signal. 
Remarkably, it is programmed to keep the frame rate constant. 
That is, if transmission losses cause the frame rate to fall, the 
receiver end duplicates properly the available frames. 

At the same time and executed in a separate thread, the 
packet-sniffer saves concurrently a network trace. So that 
whenever the transmission is completed, the following files 
are provided: 

. The network trace (PCAP file [20]). 
« Received compressed video (probably impaired with 
losses). 

« Received uncompressed video (probably impaired with 
losses). 

• Reference video (from the local database). 

At this point, the client side performs the offline analysis. 
From the RTP trace, we extract the following QoS parameters: 

« Packet size. 

« RTP sequence number. 

« RTP timestamp. 

« Packet arrival time. 

From the coded stream, we extract the following bitstream 
parameters: 

• Frame type (I, P, B). 
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• Frame size. 

The quality metrics module receives all this information. 
Figure 2 shows a detailed overview of this module. It contains 
three submodules called meters that isolate the implemented 
metrics from the rest of the application logic. Meters keep 
track of the available metrics and they communicate with those 
metrics through a standard interface. Therefore, implementing 
a new metric is as easy as writing a new Python class 
and registering it at the proper meter, without modifying the 
application core. 

B. Supported metrics 

The QoS submodule provides the following metrics: 

• Latency {L), averaged from several {N) round-trip time 
(RTT) values. 
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(1) 



Packet inter-arrival time (A), where i?j is the arrival time 
for i-th packet. 



(2) 



Jitter (J), as described in [21], where Ri,Rj are arrival 
times and Si, Sj are RTP timestamps for packets 



Dit,j)^{R,-R,)-{S,-S,) 
= {Rj — Sj) — {Ri — Si) 



(3) 



X \D(i-l,i)\- J(i-l) 
^ j(, _ 1) + ^ ^ L (4) 



Clock skew (T), the relative offset between packet arrival 
time and RTP timestamp, where Ri is the arrival time and 
Si is the RTP timestamp for packet i. 
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(5) 



Instantaneous bandwidth (B) for packet i, where Sizcn 
is the size of packet n and N is the number of packets 



in the last second. 



B{i) ~ Sizcn 



(6) 
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Packet loss rate (PLR), where Seqn is the RTP sequence 
number of packet n and N is the total number of packets. 



1 ^ 

PLR = ^ Seqn - {Seqn^i + 1) 



(7) 



n=2 



Packet loss distribution (PLD). If we divide the trans- 
mission time in K intervals, we can apply (7) to every 
interval k. 

PLD{k) = j:^ Yl ^^9".*^- - (Seqn-i^k + 1) (8) 



n=2 



The bitstream (BS) submodule provides the following me- 
trics: 

« Video stream framing structure (at the receiver end, 
probably impaired with frame losses). 

• Reference video stream framing structure (with no 
losses). 

• Group of pictures (GOP) size, averaged from measured 
GOP sizes once atypical values has been discarded. 

GOPsize = E[X] (9) 
with X = { GOPn ■■ 

GOPn e [E[GOP] - cr, E[GOP] + cr]} 



I-frame loss rate. We count an I-frame loss when 

GOPn > E[GOP] + a 



(10) 



The video quality (VQ) submodule provides the following 
metrics: 
. PSNR. 

. SSIM index [9]. 

. MOS, mapped from PSNR [10]. 

• DIV, mapped from MOS (as proposed in [10]). 

• As an illustration, we provide ITU-T G.1070 video qua- 
lity metric [22], because it provides an example of gathe- 
ring information from other submodules, specifically the 
packet loss rate. 

IV. Experimental results 
A. Frame misalignment 

To test the effectiveness of the Video-Tester frame rate 
sustainer procedure, a particular scenario with 2 % of packet 
loss probability was chosen. The video akiyo_cif . 2 64 
[14] was encoded with H.263 (GOP size of 15) at 128 kpbs 
and 25 fps, and transmitted over UDP unicast. 600 video 
transmissions were performed. For each transmission, we get 
the received video with and without the Video-Tester frame 
rate sustainer element. 

We choose 5 frame positions (70, 120, 170, 220 and 270 of 
299 total frames) from the reference video in order to compare 
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Fig. 3. Frame misalignment test. 
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Fig. 4. Frame misalignment test (once the videos without the first frame 
were removed). 



misalignments through all the tests. Figure 3 shows the cu- 
mulative frequency of the position occupied by the reference 
frame. The received video presents severe misalignments in 
absence of the frame rate sustainer Instead, our method highly 
increases the probability that the frames are aligned. However, 
note that there is an anomaly at position —15. 

This anomaly can be explained attending to the GOP size. 
When the loss occurs at the first frame of the first GOP (an I- 
frame), the remaining frames (P or B) have no reference to be 
decoded. Therefore, the loss of this first frame implies the loss 
of the whole GOP. To the receiver, the video transmission starts 
one GOP later, so the frame rate sustainer has no work. This 
point can be proved with Figure 4. It shows the same analysis, 
once the videos without the first frame were removed. 

Accordingly, in order to do reliable Full-Reference calcu- 
lations (like PSNR) with our frame adjustment method, we 
can dismiss the first GOP if the received video lacks the first 
frame. 



Fig. 5. MOS ratings measured in scenarios with different packet loss 
probabihty. 



B. Exemplary tests 

A bank of tests was designed in order to show the kind of 
results that can be achieved with our Video-Tester tool. The 
video akiyo_cif .2 64 [14] was encoded with H.263 at 300 
kpbs and 25 fps, and transmitted over UDP unicast. 10 tests 
were launched with increasing packet loss probability, from 
0.1 % to 3.7 %. Within each test, we measured the I-frame 
loss rate and the percentage of frames with certain level of 
MOS. 

Results are shown in Figure 5. Each stacked bar belongs to 
one test. The bottom x-axis shows that tests are ordered from 
left to right with increasing packet loss probability. The top 
X-axis shows the measured I-frame loss rate for each test. As 
expected, we can observe a progressive deterioration of both 
I-frame loss rate and MOS ratings. 

V. Conclusion 

This paper proposes Video-Tester, an extensible and 
reusable single framework for video quality assessment over 
IP networks. On the basis of two end-points (public or private 
IP addresses), it comprises all the procedures involved in 
video over IP communications. More specifically, it accounts 
from capturing to rendering video, which involves encoding, 
sending, receiving and decoding the video signal. 

Video-Tester estimates the proper parameters and charac- 
teristics required for the overall system evaluation, and due 
to its modularity, it can evaluate the impact of any of the 
components — at any level — on the final video quality. 

Notably, Video-Tester is a Python-based tool and, hence, it 
benefits from Python key distinguishing features. It also uses 
the valuable GStreamer library [13]. More precisely, to achieve 
the greatest flexibility and for making it maximally configura- 
ble, a GStreamer RTSP server [19] is designed. Moreover, a 
procedure to solve the frame misalignment problem in lossy 
scenarios is proposed. 

Summing up, Video-Tester includes the main features of 
previous works and adds further improvements in terms of 



usability, extensibility, codec support, support of transmission 
methods and metric robustness (frame alignment) in case of 
losses. The wide range of extracted parameters allows the 
implementation of virtually any kind of video quality metric: 
Full-Reference, No-Reference or Reduced-Reference metrics; 
as well as any level picture-, packet-, bitstream-based or even 
hybrid metrics. 
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